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Abstract 


Coding  capacity  is  obtained  for  the  discrete-time  additive  Gaussian  channel, 
and  upper  bounds  on  capacity  are  obtained  for  a  class  of  nonGaussian  channels 
The  results  apply  to  channels  with  or  without  memory,  stationary  or 
nonstationary.  An  assumption  is  required  in  order  to  obtain  these  results; 
this  assumption  is  appropriate  for  channels  without  memory  using  an  average 
energy  constraint  and  for  a  large  class  of  channels  with  memory. 


Introduction 


Coding  capacity  for  block  coding  of  the  discrete-time  additive  Gaussian 
channel  is  one  of  the  oldest  problems  in  the  Shannon  theory.  Nevertheless,  the 
solution  has  been  obtained  only  for  the  simplest  case:  when  the  channel  is 
memoryless,  with  constant  noise  covariance  and  with  a  simple  energy  constraint 
on  the  code  words . 

This  paper  gives  a  solution  for  the  deterministic  coding  capacity  of  this 
channel.  The  results  hold  for  channels  with  or  without  memory,  stationary  or 
nonstationary,  and  provide  upper  bounds  for  the  capacity  of  a  class  of 
nonGaussian  channels.  They  are  obtained  under  an  assumption  on  the  relation 
existing  between  the  noise  covariance  and  the  constraint  covariance;  it  will 
be  seen  that  this  assumption  is  quite  appropriate  for  memoryless  channels  with 
an  average  energy  constraint  and  that  it  will  hold  for  a  large  class  of 
channels  with  memory. 

The  channel  is  described  as  follows.  The  noise  is  a  zero— mean  stochastic 

process  {N^,  t  =  1,2,...}  defined  on  a  probability  space  (n,/3,P)  and  having 

paths  (w.p.  1)  in  (the  space  of  square-summable  real  sequences).  is  a 

probability  (perhaps  not  countably  additive)  on  the  cylinder  sets  of 

defined  by  p^{x:  (x.  ,  .  .  .x.  )  e  A)  =  ?{co:  (N.  (m) . N  ((.))  e  A},  defined 

In  '*’1 


n 


for  any  n  >  1,  any  Borel  set  A  in  and  any  set  of  integers  i.,..,i  .  (N  ) 
is  additive  noise,  determines  a  bounded,  strictly-positive,  self-adjoint 
covariance  operator  in  an  infinite  matrix  with  Rj^(i,j)  =  EN.N..  A 
constraint  on  the  transmitted  signal  will  be  given  in  terms  of  a  second  such 


covariance  operator  R^  in  A  basic  assumption  is  that  r8uige(R^)  contains 

j. 

range(R|).  This  is  necessary  in  order  that  the  information  capacity  be  finite 
[1]  and  finite  information  capacity  is  necessary  in  order  to  obtain  the  coding 
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) 


capacity.  Under  this  assumption,  there  exists  a  self-adjoint  operator  S  such 

that  =  R^(I+S)R^,  where  (I+S)  exists  and  is  bounded  (see  Prop.  1]  for 

ramifications  of  this  fact).  The  limit  points  of  the  spectrum  of  S  will  play  a 

key  role  in  this  paper.  These  limit  points  consist  of  all  eigenvalues  of 

infinite  multiplicity,  all  limits  of  sequences  of  distinct  eigenvalues,  and 

all  points  of  the  continuous  spectrum  [2,  p.  363]. 

A  simple  example  of  such  a  channel  and  constraint  is  the  memoryless 

Gaussian  channel  with  =  I  (leading  to  an  average  power  constraint)  and  R 

2  2 

given  by  Rj^(i,j)  =  with  >  e  for  all  j  >  1,  some  e  >  0.  A  more 

complicated  channel  is  considered  in  the  example  given  at  the  end  of  the 
paper . 

In  the  discrete-time  channel,  a  code  {k,n,e}  is  a  set  of  k  code  words 

Xi,..,Xj^  and  corresponding  decoding  sets  Cj,..,Cj^,  satisfying  the  constraints 

given  below,  with  the  requirement  that  each  x^  belong  to  The  decoding  sets 

are  Borel  sets  in  K^.  The  constraints  on  the  code  words  are  that  llx  11^  <  nP 

i  W,n  “ 

2  — ^  2 

where  n-dimensional  Euclidean  norm,  and  is 

the  restriction  of  R^  to  (1,2 - n}x{l,2 _ n}.  We  require  that 

n 

l^{y‘  y+Xj  €  C^}  >  1  -  e,  where  is  the  measure  on  the  Borel  sets  of 

induced  from  by  the  map  q^:  x  (x^  ,X2.  .  .  ,x^) .  R  >  0  is  an  admissible  rate 

n.R 

if  there  exists  a  sequence  of  codes  ({[e  ^  ],n.,e  })  with  e  0  as  n. 

i  ni  1 

00 

The  capacity  C^(P)  is  the  supremum  over  the  set  of  admissible  rates. 

We  will  obtain  an  exact  expression  for  the  coding  capacity  of  the 
discrete-time  Gaussian  channel  and  an  upper  bound  for  a  class  of  non-Gaussian 
channels. 

The  coding  capacity  involves  the  entropy  Hgj^(N) ,  where  p^j^  is  the 
Gaussian  noise  measure  (perhaps  not  countably  additive)  having  covariance 
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matrix  Rjj.  In  this  framework,  the  definition  is  Hgj^(N)  =  sup  ,  where 

n 


HqjjCN)  is  the  entropy  of  with  respect  to  =  J 

IR 


n 


log 


dp, 


dp 


n 


GN" 


GN 


Of  course,  Hgj,j(N)  =  «>  if  p^  is  not  absolutely  continuous  with  respect  to  p^^^. 

Bounds  on  the  coding  capacity  of  the  discrete-time  Gaussian  channel  have 
been  obtained.  It  should  be  noted  that  these  bounds  hold  without  any  further 
assumptions. 


Proposition  [3]:  Suppose  that  N  is  Gaussian.  Let  0  be  the  smallest  and  e„  the 

^  K 

largest  limit  point  of  the  spectrum  of  the  operator  S.  Then 


log 


1  + 


i+e, 


K' 


<  C5J(P)  <  t  log 


Hgn(N)  <  then  ci^(P)  <  t  log 


1  + 

1  + 


1+e 

p 


1+0, 


.  If  N  is  not  Gaussian,  and 


We  now  turn  attention  to  obtaining  the  exact  capacity.  The  basic  path  to 
be  followed  will  be  familiar  to  information  theorists,  as  Feinstein’s  Lemma 
and  Fano  s  inequality  are  applied  to  prove  the  lower  bound  and  the  upper 
bound,  respectively,  on  capacity.  However,  the  development  relies  heavily  on 
recent  results  on  information  capacity  (especially  those  of  [1])  and  on 
spectral  theory  for  unbounded  self-adjoint  operators  in  Hilbert  space  (as 
developed  in  [2]). 

In  order  to  state  and  prove  the  coding  theorem,  we  will  need  two  lemmas 
and  a  number  of  definitions. 


LemmaJ,:  Let  S:  -+  satisfy  =  r|(I+S)r|,  and  let  S^:  ^  satisfy 

J-  JL 

^.n  "  ^  1-  where  I,,  is  the  identity  in  r".  Then: 

Vn  =  Wn 
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^W.n  “  ^n^W^n*  ^n‘  ^2  is  ^  partial  isometry  with  initial  set 

equal  to  range (R^P^)  and  final  set 

(3)  S  =  V  SV’^.  n  >  1. 

^  n  n  n 

(4)  Fix  6  >  0.  Let  be  the  largest  limit  point  of  the  spectrum  of  S.  Let 

=  dim{x  €  K^:  11(1  +S  l^xll^  <  (1+0  +5)11x11^}.  Then  ^  oo  and  M^/n  1 
11  11  n  IV  n  n  n 

as  n  00^  for  every  6  >  0. 

(5)  Let  5  be  strictly  positive  and  strictly  less  than  1+  0^,  where  0^  is  the 
smallest  limit  point  of  the  spectrum  of  S.  Let  M^(0)  be  the  dimension  of 

the  set  {x  in  IR  :  <  5}.  Then  there  exists  an  integer  such 

that  M^(0)  <  Kg  for  all  n  >  1.  and  so  M^(0)/n  ^  0  as  n  -+  «>. 


Proof :  (1)  Clear. 

(2)  follows  from  (1)  and  [4]. 

(3)  Equate  the  two  definitions  of  Rj^ 

^  ^  =  r|  (I  +s  )r| 

T(,n  w.n''  n  n-'  W.n 


(a) 


j. 

>2  i 


=  R^  (V  V  +V  SV  )R2  . 

W.n'^  n  n  n  n-'  W.n 


(b) 


Since  =  I  on  range(RJP^) ,  Y^Y^YJt^?^^  =  V^RjP^x.  so  that  on  k". 

Since  =  range(R2  ),  equating  (a)  and  (b)  yields  S  =  V  SV^ . 

"•a  V  /  j  n  n 

To  prove  (4),  we  can  assume  that  0j^  <  “.  It  is  then  sufficient  to  show 
that  there  exists,  for  any  5  >  0,  Kg  <  <»  such  that  the  dimension  of 


span{x  €  >  (l+0j^+5) llxll^}  does  not  exceed  Kg. 
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Note  first  that  since  is  a  partial  isometry  with  initial  set  equal  to 


range(R|P^),  there  exists  for  any  x  in  a  unique  y(x)  in  such  that 

X  =  V  y.  Since  V  V  =  I  on  the  initial  set  of  V  ,  we  have  that 
n  n  n  xi 

dim{x  €  k":  II (I  +S  >  (l+e„+5)llxll^} 

n  n^  n  ^  K  ^  n^' 

=  dim{x  €  range(R^P^):  ll(I+S)2xll^  >  ( l+0j^+5)  llxll^} . 

Thus,  to  show  that  M  -»  <»,  it  is  sufficient  to  show  that  there  exists  K,,  such 

o 

that  dim{x  €  6^'.  ll(I+S)^xll  >  (l+0j^+6)llxll^}  does  not  exceed  K^. 

Let  {Pj^,  X  €  K}  be  the  left-continuous  resolution  of  the  identity  for  the 

operator  I+S,  such  that  the  domain  3(I+S)  of  I  +  S  consists  of  all  elements  x 

in  ^2  such  that  /“xdllP^xll^  <  «>.  with  (I+S)x  =  /^AdP^^x.  Here  the  integral 

exists  as  a  limit  of  Stielt jes-type  sums  in  the  strong  operator  topology 

(pointwise  convergence);  see  [2].  Since  0j^  <  oo,  s  is  bounded  with 
2 

<(I+S)x,x>  <  Mllxll  for  all  x  €  S  some  M  <  «>.  Suppose  that  <Sx,x>  >  0„llxll^ 

^  K 

for  some  x  in  For  any  e  >  0, 

l+0„+5  1+M+e 

I  +  S  =  XdP.  +  /  XdP.  . 

0  ^  l+0j^+5  ^ 

Now,  as  1  +  0j^  is  the  largest  limit  point  of  I+S,  the  operator 

^1+M+e  ~  ^1+0  +5  only  have  finite-dimensional  range  space.  If  the  set 
K 

2  o 

{x  €  ll(I+S)^xll  >  (l+0j^+5)llxll  }  has  infinite  dimension,  then  there  must 

exist  an  element  f  such  that  <(I+S)f,f>  >  (l+0„+5)llf  11^  and 

K 

“(''l+M.e  - 

K 

l+M+e  1+0+5 

Since  (I+S)f  =  X  XdP  f,  this  gives  (I+S)f  =  XdP.f 

0  ^  0  ^ 
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so  that 


i+e„+6 

<(I+S)f.f>  =  S  Xd<P  f.f>  ^  (l+e  +5)  d<P.f.f> 

0  ^  ^  0  ^ 

1+M+& 

<  (i+e^+6)  S  d<p,f,f> 

K  0  ^ 

or  <(I+S)f.f>  <  (l+0j^+5)llfll^. 

This  contradiction  establishes  the  existence  of  the  previously— defined 
Kg  <  00  for  every  6  >  0,  and  thus  establishes  that  oo  for  every  5  >  0.  To 

see  that  ^ that  when  n  >  Kg.  then  >  n  -  Kg,  and  so 

“n 

^  -  IT  ^  ^  giving  1. 

Part  (5)  is  proved  in  the  same  way  as  (4),  since  the  projection  operator 
Pg  ~  Pq  has  dimension  Kg  for  some  non-negative  integer  Kg.  □ 

We  give  several  additional  definitions.  The  operators  S,  S  P  V  and 

n’  n’  n’ 

are  defined  as  above.  We  shall  assume  that  S  has  K  limit  points  for  its 
spectrum,  denoted  0^  <  ©2  <  . . -  <  0j^.  Of  course,  0^  >  -1  since  (I+S)~^  is 
bounded . 

The  restriction  that  the  spectrum  of  S  has  a  finite  set  of  limit  points 
is  primarily  meaningful  in  the  case  of  the  channel  with  memory.  It  means  that 
S  has  no  continuous  spectrum  (except  possibly  zero  as  a  limit  point  of 
sequences  of  distinct  eigenvalues),  and,  in  particular.  S  has  a  complete  set 
of  eigenvectors.  For  the  memoryless  channel  with  an  average  energy  constraint, 
the  assumption  is  very  minor.  There,  it  means  only  that  the  noise  covariances 
do  not  have  an  infinite  set  of  limit  points — certainly  reasonable.  The 
assumption  will  also  hold  for  a  very  large  class  of  channels  with  memory. 

Let  5  >  0  be  such  that  5  <  t  min[0.  +  0._t],  where  0_  is  defined  as 

1<K  1  1  ^  U 

partition  the  interval  (0,  III+SII]  into  the  intervals  (0,  l+0^+6]. 
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(l+0^+5,  1+02+5],  ...  (l+0^_^+5,  1+0J+6],  ...  (l+0j^+6,  III+SII].  The  interval 

is  defined  only  when  l+0j^+5  <  III+SII.  If  S  is  not  bounded,  then 
III+SII  is  not  defined,  and  0^,  =  «>.  However,  we  will  at  times  use  III+SII,  under¬ 
standing  that  this  is  equal  to  +“  if  S  is  not  bounded.  The  spectrum  of  the 
operator  I+S  is  contained  in  the  union  of  these  K+1  (or  K)  intervals.  For 

fixed  6,  we  now  denote  by  M^(i)  the  number  of  o.n.  eigenvectors  of  I  +  S 

n  n  n 

corresponding  to  eigenvalues  which  lie  in  the  interval  (1+0.  ,+5,  1+0  +5] 

1-1  i 

P j  <  ^2  ^  •••  ^  will  denote  the  eigenvalues  of  S^,  repeated  according  to 

their  multiplicity.  For  1  <  j  <  K,  (a^’^’^,  j  <  M^(i))  will  denote  the 

sequence  of  eigenvalues  of  S  lying  in  the  interval  (0.  ,+5,  0.+51,  ordered  bv 

n  1-1  1 

n,5.i  .  n,6,i  ,  ,  ,  , 

<  °^j+l  ’  repeated  according  to  their  multiplicity. 

Let  ^  be  the  projection  operator  in  with  range  equal  to 


range(R^P  ).  (P^  )  is  a  monotone  non-decreasing  sequence  and  P^  x  -t  x  for 

all  X  in  ^2:  this  is  obvious,  since  we  can  identify  range(P^)  with  the 

subspace  =  (x  C  ^  :  x  =  0  for  i>n}.  If  x  €  range(P  ),  then  V^V  x  =  x; 

w  f  n  n  n 

hence,  (V^V^)  converges  to  the  identity  in  the  strong  operator  topology: 


V^V^x  -+  X  for  all  x  in 


Lemma  2:  For  any  5  >  0  such  that  6  <  2  min  [0. 


i<K-l 


i+1 


Oi]. 


K  M"(i) 

(1)  lim  2  — -  =  1; 

n 

n-+»  i=l 


.5,. 


(2)  lim  M^(i)  =  «>  for  all  i  <  K; 

n 

—  “n(^) 

(3)  lim  — —  is  independent  of  the  value  of  6,  for  all  i  <  K; 


n 


n 


(4)  lim  2 
n  i=l 


J  M;;(i) 


n 


is  independent  of  6  for  any  J  <  K; 
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(5)  For  £iny  5  >.  0, 


-  1  J 

lim  —  2 
n  j=l 


M 

n 

2 

i=l 


(j) 


(a 


,n.6,  j 


0.)  exists  and  is  equal  to 

J 


zero. 


Proof:  Before  proving  the  individual  results,  the  relation  between  the 

eigenvalues  of  and  the  spectrum  of  I  +  S  will  be  described.  First,  it 

is  noted  that  has  a  complete  set  of  eigenvectors,  for  all  n  >  1.  The 

assumption  that  S  has  only  a  finite  set  of  limit  points  in  its  spectrum  means 

that  S  has  a  complete  set  of  eigenvectors.  Suppose  now  that  {y.,  j<J}  is  a  set 

J 

of  o.n.  eigenvectors  for  I  +  S  ,  with  (I  ■*’S  )y.  =  a.y,.  Since  the  operator  V 

^  “  J  J  J  n 


^  Tl 

is  an  isometry  of  range(R|P^)  onto  IR  ,  we  have  that  y.  =  V  u..  where  {u . ,  j<J} 

J  ^  J  J 


IS  a 


unique  o.n.  set  contained  in  range (R^P  ).  Thus,  a  =<fI+S)v  v>  = 

"  n  J  '•  n  n-'-^^j’  n 

<V^(I+S)VnVnUj,  V^Uj>^  =  <{I+S)Uj,  Uj>.  In  particular,  0  <  l+/3j  <  ...  <  1+J3^  < 
III+SII  for  all  n  >  1. 


To  prove  (1).  we  can  assume  that  <  <».  Then,  if  the  equality  is  not 
true,  there  exists  an  infinite  sequence  (nj^)  of  integers  such  that 

>  £  >  0,  where  is  the  number  of  eigenvalues  (counted  according  to 
their  multiplicity)  of  I  +  S  that  either  exceed  1  +  0„  +  5  or  else  are 

less  than  6.  Applying  part  (5)  of  the  preceding  lemma,  we  can  assume  that  all 

these  eigenvalues  exceed  1  +  0j^  +  5.  By  the  preceding  discussion,  there  then 

exists  for  each  k  >  1  an  o.n.  set  {u^ . u^  }  with  <(I+S)u^.  u^>  >  1  +  0^  +  5 

^  i  i  K 

for  i  i  Since  1  +  0j^  is  the  largest  limit  point  of  the  spectrum  of  I  +  S, 
there  must  exist  an  integer  Nj^  such  that  qj^  <  Nj^  for  all  k  >  1.  This 
contradiction  establishes  (1). 

(2)  Suppose  that  there  exists  some  integer  such  that  dim  span{x  6 

f  ^  <M.  foralln>l,  where  =  (1+0.  .+5,  1+0. +6'], 

xinrii  1 
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Gq  =  -1.  Then  dim  span{x  €  range(R2p^):  <(I+S)x,x>  €  A^}  <  M.  for  all  n  >  1. 

Now  range(R2p^)  =  range(P^  defines  a  non-decreasing  family  of  subspaces, 

with  P^  ^x  X  for  all  x  in  and  the  above  inequality  is  equivalent  to 

dim  span{x  €  <(I+S)P^^^x,P^^^x>  €  A^}  <  M.  for  all  n  >  1 .  As  1  +  0.  is  a 

limit  point  of  the  spectrum  for  I  +  S.  there  exists  an  infinite  o.n.  set  (u^ 

^  k* 

k  >  1}  with  <(I+S)u^,  ii^>  1  +  0^  as  k  CO.  Hence,  if  we  take  +  1  of  the 

elements  snd  any  e  >  0,  then  there  exists  N(e)  such  that 

i 

<(I+S)P^_^Uj.  e  (l+e.-e,  l+e.+e)  for  j  <  M.  +  1  and  n  >  N(fc).  We  can 

assume  that  HP^  ^^Ujll  >  0  f or  j  <  M .  +  1 .  To  prove  (2).  it  suffices  to  show 
that  for  all  sufficiently  large  n  the  elements  {P„  u\,  j  <  M.+l}  must  be 

linearly  independent.  If  this  is  not  true,  then 


M. 

1 


P  u 
W,n  M.+l 
1 


n ,  i  yj 

2a.  P^. 


1 

+  TtT  U  . 

J  W,n  J 


(^) 


for  infinitely  many  integers  n.  We  show  that  this  cannot  hold. 

Suppose  that  holds  for  infinitely  many  integers  n.  Then,  as  fP  1  is 

^  W,n'^ 

a  monotone  increasing  family  of  projection  operators,  we  cttn  see  that  if 


M. 

1 


'’^W  m^^M  +1  “  ^  a'^u'?’^)ll  >  0  for  all  scalar  sets  {a*?,  j  <  M.},  then  this 

i  j=l  «J  J  J  1 

also  holds  with  n  in  place  of  m,  n  >  m.  For,  P„  =  P„  0P„  +  P„  ,  where 

n,n  w,n  W.m  W,m 

^W,n^W,m  projection  onto  the  orthogonal  complement  within  range(P^ 


of  range(P  ).  Then 
tT  ,  m 


M. 

1 


n  iM,2 


"’’w,n(^M.+l  .Vj) 

1  j=l 
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M. 

1 


n  iM,2 


M. 

1 


n 


1  J  =  1  1  J  =  1 


M. 

1 


Thus,  if  the  set  {P^  j ’  linearly  dependent  for  infinitely 

many  n,  then  it  must  be  linearly  dependent  for  every  n.  This  gives 


M. 


inf  IIP  uj  .1-2  a”P„,  u^)ll^  =  0 
[a"?.  j<M.}  J  W.n 


for  every  n.  This  implies  that  for  every  n  there  exists  fa.,  i  <  M  }  such  that 

J  i^ 

Mi 

the  vector  ^.i  “  2  a^u^  is  contained  in  range(P^  ).  Since  (P;^  )  is  a 

^  i=l  n  t n.  WjH 


M. 

1 

monotone  decreasing  family  of  projections,  this  means  that  I  (a^-a^^u^  is 

i_i  J  j  j 

contained  in  range(P  )  when  n  <  m.  If  a"  ?£  a"*,  then  as  the  {u^,-  j  <  M  }  is 

w.n  J  J  I  J  u  _  jj 

an  o.n.  set,  u^  must  belong  to  range(P^  ).  If  u\  is  in  range(pi  )  infinitely 

often,  then  necessarily  u^  is  in  range(P;^  )  for  every  n  (because  (P,-|:  )  is  a 

J  n.n  W.n^ 

monotone  decreasing  family  of  projections).  This  contradicts  the  fact  that 
<(I+S)P^^^Uj,  is  in  (l+G^-e,  l+G^+e)  for  all  n  >  N(e).  We  thus 

conclude  that  =  a™  for  all  but  a  finite  set  of  integers  n,m.  Hence, 

J  J 

n^ j ’  ^  -  ^i^  linearly  dependent  for  infinitely  many  n  only  if  then 

M. 

1 

there  exists  N  such  that  P^  =  _2  a^P^  ^uj  for  some  fixed  set 

{“•.  j  ^  M. },  all  n  >  N(e).  But  then 

J  ^ 


M. 

i  ^  i  2 

0  =  lim  IIP„  (u„  ,  -  2  a.u.)ln 

^  W,n'-  M.+l  .  1  J 

n  1  j=l 

Mi  M. 

=  "“ill  - 

1  J=1  ^  1  j=l 
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This  contradiction  shows  that  (h)  above  cannot  hold  for  infinitely  many 
integers  n,  and  thus  proves  (2). 

■  g 

(3)  The  fact  that  lim  M^(i)/n  is  independent  of  the  value  of  5  follows 

n 

from  the  fact  that  there  exists,  for  i  <  K.  an  integer  depending  on  5  such 

that  can  have  no  more  than  eigenvalues  in  the  interval 

(l+0i_i+5.  l+e.-5].  for  5  <  t  min[e  -  0  ]  and  for  all  n  >  1.  For,  if  this 

i<K  ^  ^  ^ 

were  not  true,  then  the  discussion  at  the  beginning  of  the  proof  shows  that 

I  +  S  would  have  a  limit  point  in  the  interval  f 1+0  +5,  1+0  -61 

^  1  i+1  ^ ’ 

contradicting  the  definitions  of  0.  and  0 

1  i+1 

(4)  follows  in  the  same  way  as  (3). 

To  prove  (5).  one  simply  notes  that  for  any  e  <  5  and  i  <  K  there  can 

exist  only  a  finite  number  of  o.n.  elements  (u.)  in  satisfying 

1  +  +  5  <  <(I+S)u.  ,u’. >  <  1  +0.  -  e. 

^  ^  111  n 

The  proof  of  the  coding  theorem  will  use  Feinstein’s  Fundamental  Lemma, 
as  modified  and  extended  by  Thomasian  (see,  e.g. ,  [5,  p.  232])  and  Kadota  [6]. 
It  is  stated  below. 

Lemma  3-  Let  ,/3^  and  (122.^2’^)  probability  spaces,  with  f: 

12^x02  fig  PjX/32/P2-n'easurable  function.  Define  on  (0^x02-  by 

M^(D)  =  p^®pj^{(x.y):  (x.f(x,y))  e  D} . 

and  let  p^  be  the  projection  of  p^  onto  (note  that  Py(C)  = 

^  D} )  •  Let  f^  be  the  section  of  f  at  x:  fj^(y)  =  f(x,y),  any  fixed  x  in 

Qj,  all  y  in  ^2.  Suppose  that  and  p^^  are  mutually  absolutely  continuous 

a.e.  dp^(x),  and  that  the  function  g:  (x,y)  [dpj^of“VdpY](y)  is  ^-^><^2 
measurable.  For  any  real  number  a  let  =  {(x,y)  in  n^xn2-  g(x,y)  >  a}. 
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Then  for  each  positive  integer  k  and  set  F  in  there  exists  a  code  (k.F.e) 
such  that  e  <  ke~“  +  +  fij^(F°). 

The  following  theorem  gives  the  exact  coding  capacity  for  the  discrete¬ 
time  Gaussian  channel.  Recall  that  we  are  assuming  WLOG  that  is  strictly 
positive  on 


Ihepr^:  Suppose  that  R^^  =  Rj(I+S)R|.  where  (I+S)  ^  exists  and  is  bounded,  and 

that  S  has  pure  point  spectrum  with  K  limit  points  0.  <  0  <  ...  <  0  .  Then, 

12  K  * 

if  Hgjj(N)  <  00. 


cJJCP)  <  t  lim  2  'Tj  log 
n  i=l 


where  — with  5  any  fixed  number  in  (0,  ^  min[0.  .-0.]).  and  J<K  is 

i<K  ^  ^  ^ 


the  largest  integer  such  that  P  +  lim  2  t”(0.-0  )  >0.  If  N  is  Gaussian,  then 

n  i<J  ^  ^ 


Cm(P)  =  t  lim  2  nr”  log  ^ - 

n  i=l  L(2J^^'r")(l+0.)J 

Proof:  The  proof  is  rather  lengthy,  but  follows  a  standard  path.  First,  we  use 
Feinstein’s  Lemma  to  prove  that  C^(P)  as  given  is  a  lower  bound  when  N  is 
Gaussian.  Fano’s  inequality  is  then  applied  to  show  that  it  is  an  upper  bound. 
The  results  of  [1]  and  [8]  are  essential  for  these  proofs.  Notice  that  we  can 
assume,  by  the  Proposition,  that  0^^  <  oo. 

We  will  use  Lemma  3  to  prove  the  lower  bound  when  N  is  Gaussian.  That 
lemma  first  requires  one  to  define  a  Gaussian  probability  measure  and 
thereby  and  p^0p^,  for  n  >  1.  such  that  -»  0  and  ^  0.  where 

d  ^ 

F  =  {x  €  r":  IIxIiJ  ^  <  nP} .  =  {(x.y)  in  rV:  log  — ^  (x.y)  >  a  }. 
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Let  Q  €  (O.P).  Fix  5  >  0  such  that  5  <  t  min[0.-e._  ]  (with 

i<K  ^  ^  ^ 

J(Q) 


~  J(Q)  largest  integer  such  that  Q  +  lim  2  't^CG  -0  ,  ,-61 

n  J=1  J  J  Jt«) 

J(Q)  „ 

IS  strictly  positive.  For  n  such  that  Q  +  2  ^.(0.-0.  -6)  <  0,  set  =  0, 

J  J  Jl.^'  k  ’ 

k  <  K.  Otherwise,  set 


-.n 


Qi  = 


J(Q)  „ 


k  i  J(Q) 


=  0  ' 


k  >  J(Q), 


Let  be  the  zero  mean  Gaussian  measure  on  with  covarieince  operator 
(matrix)  given  by 


r,n  ''^n  ,,  n.i- 


where  {U  u  .  1  <  j  <  M  (i),  1  <  i  <  K}  are  o.n.  eigenvectors  of  I  +  S 

“  n  n 


with  <(VSn)uV’\  €  (l+0^._^+5.  1+0J+5].  U^:  ]R^  ^  k"  is  the  unitary 

matrix  satisfying  =  R|  inner  sum  in  the  definition  of  R^ 

is  only  defined  for  those  i  such  that  M  (i)  y  0.  Note  that  u?  =  l£  o  fR^  ^ 

n'^  ^  ^  ^  N,n^  ’ 

where  is  the  zero-mean  Gaussian  measure  on  with  covariance  operator 

,  J(Q)  .  . 

R?=  2  q"  2  u’?’J®u’?’J. 

^  j=l  J  i=l  ^  ^ 

Let  =  (x  €  K^:  llxll^^^  <  nP} .  We  will  show  that  pJCF^]  ^  0  as  n 

1^[fJ]  =  li5{x  €  ||xl|2^^  >  nP} 

=  fLj{x  e  Ir'^:  IIr|^^xII^^^  >  nP} 
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=  M?{x  6  Ir”:  >  „P> 


r.n. 


=  |ii?{x  e  k":  II(I  +S  )2u^xil^  >  nP} 

1  n  n  n  ^ 

n  n  X  n  9 

=  |J^{x  e  K^:  2  11(1  +S  )2U  P^xir  >  nP} 

i  '  n  n-'  n  j  n  ^ 


M^(J) 

where  p”x  =  2  <u?’ ,x>u?’ .  Thus, 

•^1=1^  ^ 


J(Q) 


M^(J) 


^vCFn]  <  )jS{x  e  K^:  2  (1+e  +6)  2  <u?’'^  .x>^  >  nP} . 

j=l  ^  i=l  ^ 


For  each  fixed  j  <  J{Q)  such  that  q"  >  0.  the  set  of  random  variables 

%} 

i  ^  ^  zero-mean  independent  Gaussian  random 

variables,  each  having  variance  q’^,  with  respect  to  Moreover,  <p?’'^,  •>  is 

J  1  i 

independent  of  ,•>  for  j  ?£  j',  all  i  <  M^(j),  all  k  <  M^(j').  Let 

J(Q) 


M^(j) 


Z^=  2  (1+0 ^.+5)  2  <p^”^ 


j=l 


2  <p. ,  and  set  t.  =  1  +  0.  +  5,  v 

1=1  1  j  j  1 


"■J  =  <k"-J,->2. 


Then 


S  n 

E  Z  =  2  T.M  {J)Q.. 

J=1  ^ 


Also 


2 

=  2  2  T.T. 

""  j=l  i<J(Q)  ^  ^ 


'■m^(J) 

2  v 
k=l 


n.  j 


'M^(i) 
n^  ■' 

2  V 

e=l 


n,  1 


J(Q)  2 

+  2  T  . 

j=l  ' 


'■mJ(j) 

2  v' 
k=l 


n,  j 


Computing  E  Z  ,  one  obtains 
n  n 
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O  J{Q)  r  sr 

E  TT  =  2  2  T.T.M'^(i)M°(j)Q”Q^ 

"  j=l  i<J(Q)  ^  ^  ^  ^  J 


J(Q) 

+  2  r't 


J=1 


3(Q';)V(j)  +  M^(j)[M^(j)-l](Q^)2 


J  L  J  n 


J(Q) 


=  2  ^ 

•  1  J  n  J  n  1 


i<J(Q) 


2  J(Q) 


The  variance  of  with  respect  to  p^is  thus  equal  to 


j=l  J  y 


Applying  Chebyshev’s  inequality,  we  obtain 
'4<Z„  >  nP>  =  ^{Z^-E  >  „P-E 

Mj  Mj 


Now, 


(nP-E  Z  j=l  J  ^  J' 


(^) 


J{Q)  5 

E  Z  =  2  M  (j) 
Uj  J=1 


J(Q) 


Q  +  2  'r"[0.-0  -5] 

1  J  -' 


i=l 


J(Q) 

2  -Y 

k=l 


n 


J(Q) 


=  nQ  -  n  2  >  n(Q-5) 

j=l  ^ 


(^) 
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Similarly,  the  variance  of  Z  is 

n 


2,  5  n  2  '^(Q)  g 

2  2  T^^j)(QV  =  22  M^(j) 

j=i  ^  ^  ^  j=i  ^ 


■  n 

Q  +  .2^  ^i[0i-ej-6] 


J(Q) 

2  T 

k=l 


n 


J(Q)  „ 

=  2n  2  -r  . 

j=l 


J(Q) 


Q  +  2  'r^ie.-e-51 

1=1  1*- 1  J  ^ 


J(Q) 

2  t: 
k=l 


n 


J(Q) 


2n(Qf0j^)' 


J(Q) 

2  T 

k=l 


n 


To  see  that  2  -r^  is  bounded  away  from  zero  for  all  sufficiently  large 

n,  suppose  not.  Then 


tind  so 


lim 

n 


J(Q) 


J(Q)+1 


=  Q 


lim  [q  .  _2^  -i(ei-ej(Q)..r5) 


=  lim 
n 


Q  +  T 


J(Q)+l^®J(Q)+l  ®J(Q)+1 


=  Q  -  1^  ^5(Q)+1^- 


J(Q)+1 


Now.  lim  .  does  not  depend  on  5;  thus,  if  lim  2  >  0  while 

n  n  i=l  ^ 


J(Q) 


—  "^i  ^  —  ^J(Q)+1  ^  and  so  Q  -  '’^5(0)+!^  >  Q  -  5  for  all 


—  r  n  1 

sufficiently  large  n.  Taking  5  <  Q,  lim  Q  +  2  'rJe.-Orr^^  --5) 

n  L  i=i  1  J(Q)+1 

inequality  contradicts  the  definition  of  J(Q)  if  Hm  >  0,  showing  that 

n  'Jvyj+l 


>  0.  This 
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J{Q)  „ 

1  y  is  bounded  away  from  zero  for  all  sufficiently  large  n.  Noting  that 

j=l 

^  n  n 

lini  2  -Y  =  1,  we  see  that  lim  2  t  .  >  0  is  required  by  the  definition  of 


n  j=l  ^ 

J(Q)- 


n  j=l 

Using  (^)  and  in  (^) ,  one  obtains 


n 


n 


2(Q^-ej^r  1 

n(P+5-Q)2 


which  converges  to  zero  as  n  -»  <»,  for  all  Q  <  P,  any  5  in  (0,  t  min[e.-0._  ]) 

i<K  ^ 

We  now  proceed  to  verifying  the  next  step  in  applying  Feinstein’s  Lemma: 
For  as  defined  above,  and  the  resulting  N^[A^]  ^0,  with 

dfi^ 

=  {(x.y)  in  IR  xR  :  log  — — —  (x.y)  >  a^}. 


J(Q) 


-  2  2  2  log[l  +  Q,.]  “  mr,  where  t  >  0  is  fixed. 

j=l  i=l  ^ 

Applying  the  proof  of  Proposition  2  of  [8],  we  have  that 

J(Q) 

A^  =  {(x.y):  t  2  2  <  -ny} 

j=l  i=l  ^  ^ 

where  for  j  <  J(Q)  such  that  q’J  >  0.  {a^’-^.  bj’*^:  i.k  <  is  a  set  of 

J  1  -K  n 

i.i.d.  Gaussian  random  variables  with  respect  to  |u^,  each  having  zero  mean 


and  variance  (Q^/[l+Q^])^ .  Moreover,  the  set  of  random  variables  (a^’^.  b^’*^: 

J  J  i  k 

i.k  <  M^(j)}  is  independent  (w.r.t.  of  the  random  variables 

{a. :  i.k  <  for  j  ^  j'.  Chebyshev’s  inequality  then  gives 
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-  2  2^n 

n 


rJ(Q)  .  p  .  p  .2 

j=l  i=l  ^  ^  ■ 


1  J(Q)"nfJ) 


9  9  2  2  E 

n  -Y  j=l  i=l  )u^ 

,  J(Q)  2Q^ 

-i—  T  y  — J - i 


(a^’  '^)'^-2(a^’  •’b^’  '^)^+(bJ’  •^)^] 


j=l  1=1  k+Q"  1+q” 
J  J 


1  5  f^n  4  J(Q)  nf  Q"?  1  4 

=  2  M°(j)  — J-  =  -^  2  M-  <  -4r 

n  TT^  j=l  4+q"-I  n-r^  j=l  ^  4+q’^-'  n-r^ 


^  0. 


Now,  by  Feinstein  s  Lemma,  there  exists  a  code  (k  ,F  e  )  with  k  =  f 

^  n  n  n^'  n  '- 

~a 

and  <  k^e  +  f^(A°)  +  f^(F^)  •  It  thus  remains  only  to  determine  the 

of  R  such  that  k  e  -»  0  as  n  -^  “>.  By  definition  of  a  ,  k  e  ^  < 

'n  n  “ 

-  J(Q)  M^(j) 

.  Choosing  R  <  lim  f  2  — -  log[l+Q.].  R 

n  j=l  "  ^ 

admissible,  since  then  one  can  take  -y  >  0  such  that 

J(Q)  5 

nR  +  n-Y  -  f  2  M^(j)  log[l+Q^]  <  0  for  an  infinite  set  of  integers  n  >  1. 

j=l 

This  gives  R  admissible  so  long  as 


nR+mr-f  2'|(^)M^(j)  log[l+Q"] 

a  Jin  J 


and  so 


J(Q) 


R  <  lim  2  2  log 
n  j=l  ^ 


J(Q) 


•  n 

Q  +  2^  'TjCl+e.) 


Q  +  2  ^“(1+ep 

cJJ(P)  >  lim  t  '2  ^”  log 


n  j=l 


J(Q) 


■J(Q) 

■k 


ni  r  1 

’SJ  hjJ 
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Let  J  ^  K  be  the  largest  integer  such  that  P  +  lim  2  ^’^(e.-e.)  >  0.  Then,  for 

n  j=l  J  J 

sufficiently  small  5  >  0,  there  exists  Q  <  P  such  that 

J 


Q  +  2  'r"(e.-0  -5)  >  0 

J=1  J  J  ‘J 

for  infinitely  many  values  of  n;  for  such  5  and  Q,  we  thus  have  J(Q)  =  J.  If 

-  J  n 

P  +  lim  2  nr  (0-0  )  =  0.  then  for  sufficiently  small  5  and  for  Q  sufficiently 
n  j=l  J  J  'J 

near  P,  J(Q)  =  J-1. 

—  J  n. 


Thus,  if  P  +  lim  2  nr.(0.‘-0,_)  >  0,  then  (^)  gives 
n  j=l  ^  ^ 


C^(P)  >  lim  2  'y’?  log 
n  j=l  ^ 


J 


P+  2  'r?(l+0.) 
i=l  ^  ^ 


2  [l+0.1 

■k=l  ^ 


Suppose  that  P  +  lim  2  ^.(0.-0)  =  0.  Then 

n  j=l  ^  ^  -J 

J  _ 


lim  2  TT^  log 


P+  2  'rV(l+0.) 
i=l  ^  ^ 


J-1 

<  lim  2  log 


J 

P+  2  T 
i=l 

i(l"®i)' 

since  for  any  e  >  0, 


+  e 


for  all  but  a  finite  number  of  values  of  n.  We  will  now  show  that  for  every 
£  >  0,  all  but  finitely  many  n. 


OC  of  D-T  Chnls.  -  11/10/87  -  19 


J-1  „ 

2  TT  .  log 

j  =  l  ^ 


P+  2  'r^(l+e.) 
i=l  ^  ^ 


T, 

<  2  Tr  .  log 


1&]K] 


j-i 


P  +  2  -r^fl+e.+e) 

i=l  "  ^ 


For  a  fixed  value  of  n,  this  will  obviously  hold  if  =  0.  If  -r^  >  0.  then 

d  d 

the  preceding  inequality  holds  if 


•J-1  r, 
0  >  2 
j=l  ' 


which  is  equivalent  to 


J-l  n  d-1  „  d-1  d 

0  >  (1+e  )  2  -  P  -  2  -rjci+e.)  -  e  2  2  -rM 

k=l  i=l  "  "  i=l  ^  k=l  ^  J 


or 


•J-l  r,  ■ 
0  >  (1+0  )  2  -rj  - 
k=l 


•J-1  T, 
P  +  2  -r 

i=l  1 


^d  n 


j-i 


Since  P  +  lim  2  -y  (0  -0  )  =  0,  the  preceding  inequality  must  hold  for  all  but 
n  j=l  -J  J  •J 

_  d 

a  finite  set  of  n,  for  any  e  >  0.  Thus,  when  P  +  lim  2  T’^f0  -01=0 

n  d=l  ^  ^  ’ 


d-1 


d-1 


C^(P)  >  lim  2  log 
n  j=l  ^ 


P+  2  ^^(1+0.) 
i=l  ^  ^ 


d-1 

4c. 


fr^]  Hj]  J 


>  lim  2  log 
^  ,-_i  J 


■p+  2  -r 

i=l 

?(i-ei)- 

■ 

This  completes  the  proof  that  when  is  Gaussian 
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CC(P)  ^  lim  2  -T^  log 


n  j=l 


P+  2  'r’?(l+e.) 
i=l  ^  ^ 


J 

*iC: 


2  'rjjl  [l+0. 

:=1  ^-1  >-  J-l 


where  J  <  K  is  the  largest  integer  such  that 

_  J 


P  +  lim  2  T”(e.-e  )  >  0. 


n  j=l 


j  j 


for  {"rj.  j<K}  defined  in  terms  of  any  fixed  5  >  0  such  that 

6  <  t  min  (0  -0  ) . 

i<K 

In  order  to  prove  the  upper  bound  for  cJ(P),  Fano’s  inequality  is  applied 

in  a  standard  way  (see.  e.g. .  pp.  166-167  of  [8]).  Thus,  if  we  have  a  code 

nR- 


(k^.n.e),  with  =  [e  ].  then 


e  >  1  - 


C^(nP)  +  +  log  2 


log  k 


n 


This  gives  C^(P)  ^  lim  ^  C^(nP) ,  where  C^(nP)  is  the  capacity  of  the  discrete¬ 
time  Gaussian  channel  of  dimension  n  and  with  the  constraint  E  llxll^  <  nP 

n  W.n  ~ 

We  now  evaluate  lim  ^  C^(nP) .  From  Theorem  1  of  [1], 
n 

^  N(n)  rnP  +  2^^^^(l+j3"), 

cJCnP)  =  t  2  log  I - 


i=l 


N(n)[l+p”] 


N(n) 


where  N(n)  <  n  is  the  largest  integer  such  that  nP  +  2  js”  >  N(n)/3|^,  We 

i=l  ^ 


can  rewrite  this  as 
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cJCnP)  = 


K(n) 

2  2  2  log 

j=l  i=l 


MfCk)* 


P  + 


j  K(n)  f  . 

-  J  2  [l+aj-'®’'' 

k=l  e=l  L  «  J 


Nlni 

n 


where  0  <  5  <  2  n'in[0.^^-0  ] ,  K(n)  is  the  largest  integer  <  K  such  that 
i<J 

^N(n)  ^  ^^^K(n)-1^^'  number  of  elements 

£  <  M^(i)^}  in  the  sequence  (P^ . ^(n)^ 

interval  (0._j+5.  0.+5]  (0^  =  -1).  The  difference,  if  any.  between  {M^(i)^, 
i<K}  and  {M^(i).  i<K}  as  previously  defined  can  arise  only  for  i  >  K(n) . 

Considering  lim  ^  C^(nP) .  it  is  clear  that  we  can  replace  K(n) .  in  the  above 
n 

expression  for  C^(nP).  by  J:  the  largest  integer  i  <  K  such  that 

^N(n)  ^  infinitely  many  n.  J  can  also  be  characterized  as 

the  largest  integer  <  K  such  that  nP  +  2*!  >  suola’^’ ^ : 

j=l  1=1  1  - 

i  <  ^  inf initely-many  values  of  n.  Thus. 


S{p) 


J  <«)“ 


<  2  lim  —  2 


n 


n 


j=l  i=l 


log 


P+^  2  2  |l+a5’^’^l 

^k=l  5=1  t-  ^  J 


Now.  for  any  fixed  n. 


J  .<(«" 
2-2  log 
j=l  "  i=l 


Mf(k)^ 


^k=l  £=1  ^  ^  J 


is  independent  of  5.  since  the  sequence  i  <  M^(j)^.  j  <  J)  is 

3  ^ 


identical  to  the  sequence  (j3^,  i  <  N(n))  for  every  n. 


(^) 
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for  all  but  a 


Note  that  if  there  exists  K,  <  “  such  that  <  K, 

J  -  J 

finite  set  of  integers  n,  then 


J-l  “n'J) 


C^(P)  <  t  lim  i  2  2  log 

j=l  i=l 


n 


1  J-l 

P  +  —  2 

^  k=l 

«=1  i-  «  J 

Nisi 

n 

I  1  J  J 

,  ,N(n)  «n(J) ,  „ 

I'n  “  ^  — I  0  as  n  ^  00.  Thus,  we  can  assume  without  loss  of 

j=l 

generality  that  oo  as  n  oo. 

n  ' 

Let  k  <  J  and  take  any  e  >  0,  Then  there  exists  N(e)  such  that  at  most 
N(e)  of  the  elements  (a^  ),  5  <  M^(k)  ,  lie  outside  the  interval 

.  with  N(e)  independent  of  the  value  of  j.  Hence 


J 

I  ^  %  r  n.5,k__  , 

-2  2  [a  -e  ] 

k=l  5=1  ^  ^ 


e  i  ^  N£el_  ^ 


5  ^ 

2  M  (k)  /n  <  1,  we  thus  have  that 
k=l  ^ 


lim 

n-joo 


T  M  (k)  T  «5, 

1  ^  ^  r,  ^  u  J  M  (i) 

^2  2  -  2  -ii— 0. 


k=l  5=1 


i=l 


n 


(XX) 


exists  and  equals  zero. 

Using  this  fact,  the  preceding  upper  bound  on  C^(P)  can  be  written  as 


J 


UU  ^  ~  V  w  y 

C^(P)  i  2  lim  2  log 

n  j=l  ^ 


J  M^(i) 

P  +  2  — - (1+0.) 

i=l 


J  Mf(k)^ 


t=l  ^  J 


1+e. 

jj 


A  routine  calculation  shows  that  for  0  <  ,  i  <  J, 
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J 

2  a  log 
j=l  ^ 


J 

■ 

p 

+  2  a. (1+0.) 

i=l  ^  ^ 

^  1 

r  1 

. 

^  “k 

J 

is  a  monotone  increasing  function  of  a^.  Thus,  we  can  replace  M^(J)^  with 


n.5  „6,  . 


define  nr"’''  =  M"(j)/n.  and  recall  (Lemma  2)  that  lim  =  lim 

J  n  „  J  n 

n  *'  n 

is  independent  of  the  value  of  6,  for  5  in  the  interval  (0,['2  min  (0. 

i<K  ^ 

Thus,  we  have  that 


MfCj) 


C5J(P)  <  t  lim  2 


n  j=l 


P  +  2  'r’?(l+e.) 

i=l  ^  ^ 


The  final  step  is  to  show  that  J,  as  defined  in  this  part  of  the  proof, 
is  the  largest  integer  <  K  such  that 


lim 

n 


P  +  2  'r^(e.-e  ) 


j<J 


j  j  J\ 


>  0. 


We  have  defined  J  to  be  the  largest  integer  <  K  such  that 

J  ”n(J)" 

(a)  nP  +  2  2  >  N(n)  sup{aj’  i  <  ^  infinitely  many 


n,  and 

(b)  CO. 

Now  lim  sup{a^*^’'^:  i  <  M^{J)^}  >  0  and 
n  ^ 

I?  -  0  -  n  -  Thus. 
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lim 

n 


^  that  J  <  Jj,  where  is  the  largest  integer 

_ r  Ji 


<  K  such  that  lim 


1  n. 


im  P  +  2.  -nr  .(0.-0-  ) 
n  L  J=i  J  Jf 


>  0. 


We  will  now  show  that  if  J  <  ,  then 


J 


lim  2  log 


p  +  2  x(i+e, ) 
k=l  ^  ^ 


'1 


<  lim  2  log 


p  +  2^;;(i+0 ) 

k=l  ^  ^ 


(Jew) 


This  will  be  shown  by  constrained  optimization.  Define  the  function  F(a), 


a  €  K'^ ,  by 


F(a) 


2  a. 
i=i  " 


log 


J 

P  +  2  a. (1+0.) 

.1=1  J  J 


We  will  maximize  this  function,  subject  to  the  following  constraints’ 

(A)  0  <  a  <  i  <  J. 

3  ^ 

(B)  2:Ja.  <  1. 

The  inequality  (Jew)  above  will  be  proved  if  one  can  show  that  F  has  a 
global  maximum  (subject  to  the  constraints  (A)  and  (B))  given  by  a.  =  t?, 
i  <  J. 

First,  we  note  that  the  admissible  values  of  a  constitute  a  closed  convex 

set  in  IR  •  if  cl  and.  q  are  both  admissible,  and  0  ^  X  ^  1,  then  clearly 
Xa^  +  (l-X)p.  €  [0,Tr^]  and  2^[Xa.  +  (l-X)p.]  €  [0.1]  for  i  <  J.  Moreover,  this 
convex  admissible  region  is  closed.  Thus,  any  local  maximum  of  F  will  be  a 
global  maximum  if  F  is  concave  over  the  set  of  admissible  a  [9].  We  also  note 

that  the  admissible  region  is  bounded  from  below  and  that  the  global  maximum 
of  F  over  this  region  must  be  finite.  Thus  if  F  is  concave,  then  F  takes  on 
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its  maximum  at  a  boundary:  i.e.,  for  i  ^  J.  either  a.  =  0  or  Thus,  to 

prove  we  need  only  show  that  F  is  concave,  and  that  the  constrained 

optimization  problem  has  a  local  maximum  for  F(a)  given  by  a.  =  i  <  J. 

To  show  that  F  is  concave,  we  will  show  that  its  Hessian  matrix,  whose  ij 
element  is  given  by  d^T/(^da^da.^')  ,  is  negative— def  ini te.  The  calculations  give 


3^  a 

1 

2  a 

1+0.  1 

+  Ino- 

1 

CD 

+ 

r-l 

I  PN  T\ 

+ 

PU 

1 _ 

da.da .  ~  da . 

1  J  J 

J  J 

P  +  2  ^  “k- 

k=l  ^  ^  k=l  ^ 

^  w: 

J 

•  [\=l“k. 

1+0. 

i_ 

. 

J 


1+0.  1 

1 

Wial 

-(l+0p(l+0.) 

1 

J  J 

Lii  «J 

r  J  1 

2  r  J  i2 

P  +  2  a^(l+0j^)  2  CL_ 

k=l  ^  ^  k=l  ^ 

h“k 

^k=l 

1  +  0. 
L 


1 


[p  +  2  OL  (1+0  ) 
k=l  ^ 


J 


Setting  C  =  P  +  have 

d^/da^da^  =  N(  i  ,  j  )/D 
where  D  =  . 

and  N(i.j)  =  C(2;J^^aj^)(l+0.+l+0^.)  -  {l+Q  (l+Q  .)  .  To  show 

that  F  is  concave,  it  remains  to  show  that  the  matrix  with  elements  N(i,j)  is 

negative  definite.  Let  A  be  the  JxJ  matrix  whose  ij  element  is  equal  to 

)  >  for  j  ^  J,  each  i  ^  J.  A  has  constant  rows,  so  A  has  constant 

columns,  and  if  x  €  IR*^,  2^  x?  =  llxll^  =  <x.x>,,  then  S'/  .  ,x.N(i.j)x.  = 

1  =  1  J  J  l.J  =  l  1  ^  ^  J 

^2  ^  T 

-II(CIj-A  )xllj,  since  A+A  has  (2jaj^)(l+0^+l+0j)  as  ij  element.  The  matrix  N  is 
thus  negative  definite.  Since  no  restriction  has  been  placed  upon  a  in  this 

derivation,  it  follows  that  N  is  negative  definite  over  and  in  particular 
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over  the  closed  convex  set  {a:  0  ^  a  ^  ~r^ ,  T!_^a.  <  1}.  Thus,  we  have  that 

11  1  —  1  1 

any  local  maximum  for  F  also  defines  a  global  maximum. 

It  now  remains  only  to  show  that  a  local  maximum  for  F  is  obtained  for 
n  .  /  T 

“i  =  ^  ^  J- 

The  Kuhn-Tucker  objective  functional  for  this  problem  is 


J  r  T,-i  J 

f(g)  =  F(g)  +  2  |3  a.-Tr.  -  2  X.a. , 

i=l  ^  i=l  ^  ^ 


where  the  generalized  Lagrange  multipliers  and  the  vector  a 

provide  a  local  maximum  when  the  following  conditions  are  satisfied: 


da. 

1 


f(«) 


=  0,  i  <  J; 


and 


a.  -  <  0.  <  0. 

11  1  i'^  1  1^ 


=  0,  =  0,  each  for  i  <  J, 


ji  [“r']  ^  “■ 


Differentiating, 


af 


1 


■p 

+  2  a. (1+0.)' 

J=1  ^  J 

r  J  If  1 

• 

/A  H 

- 

- 


X.  . 
1 


Setting  X^  =  0  and  for  i  ^  J,  and 


-/3.  =  log 


'P  +  2  T 

j=l 

■  [Vk 

Li-'!:] 
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we  see  that  -/3.  >  0  for  i  ^  1.  This  shows  that  F(J.'r” . -Tj)  is  a  non¬ 
decreasing  function  of  J.  Hence,  above  is  proved,  and  this  completes  the 

proof  of  the  theorem. 


□ 


For  the  time-discrete  memoryless  Gaussian  channel  with  =  I.  the 

Theorem  gives  new  results  for  the  non-stationary  channel.  In  this  case. 

Rj^  =  I  +  S.  and  the  limit  points  {1+0^ . 1+0^^}  are  the  limit  points  of  the 

eigenvalues  of  R^^.  The  eigenvalues  of  Rj^  ^  are  also  eigenvalues  of  R^^.  and  the 

above  development  can  be  simplified.  However,  the  results  given  in  the 

Proposition  and  the  Theorem  for  the  capacity  C^(P)  of  course  remain  true.  In 
2  2 

the  case  of  Rj^  =  a  I.  then  a  is  the  only  limit  point  of  I  +  S,  and  so  one 

obtains  the  well-known  result  that  C^(P)  =  t  log  1  +  — 

*■  a  ' 


.  From  the 


Proposition,  this  is  also  the  value  of  cJJ(P)  if  R^  =  I  and  R^^  =  a^I  +  M,  where 
M  is  any  matrix  (not  necessarily  diagonal)  operator  in  such  that  M  is 
compact.  This  follows  from  the  fact  that  compact  operators  in  a  Hilbert  space 


are  exactly  those  operators  that  have  zero  as  the  only  limit  point  of  their 


spectrum.  Thus,  if  the  noise  is  of  the  form  N  =  +  N2.  stationary  and 

uncorrelated  with  variance  .  and  independent  of  N.  with  E  2  [N„(k)]^  < 

^  k>l  ^ 


then  the  coding  cajjacity  is  again  2  log 


or 


.  Of  course,  we  are  assuming  as 


always  that  all  processes  have  zero  mean. 

A  principal  application  of  these  results  is  to  the  arbitrarily- varying 
Gaussian  channel.  The  memoryless  GAVC  has  been  analyzed  by  Hughes  and  Narayan 
[10],  who  obtained  the  X-capacity  using  random  coding.  Using  the  results  given 


here,  it  has  been  possible  to  obtain  the  actual  (deterministic)  coding 
capacity  of  the  GAVC  [11]. 
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Example 


The  Proposition  and  the  Theorem  will  be  illustrated  by  an  example  for  the 
memoryless  nonstationary  Gaussian  channel  with  constraint  covariance  =  I. 
Let  be  the  diagonal  matrix  with  non-zero  elements  given  by 


=  1  +  i-1 

i  7^ 

integer  multiple  of  7  or  20 
k  an  integer  >  1 

=  4  +  i~^ 

i  = 

7k,  k  an  integer  <  100 

=  2  +  i"^ 

i  = 
k 

7k,  k  an  integer  >  100, 
integer  multiple  of  20 

=  5  +  i~^ 

i  = 

20k,  k  an  integer  >  1 

=  10  +  i“^ 

i  = 

2  2 

33k  ,  k  /  integer  multiple 

of  7  or  20. 

This  is  the  covariance  of  an  uncorrelated  noise  source  containing 
periodicities. 

In  this  case,  Rj^  =  I  +  S,  so  that  S  has  limit  points  9^  =  0,  ®2  ” 

bounds  given  by  the  Proposition  are 

i  log[l  +  ^  <  c5J(P)  <  t  log[l  +  P]. 


M 


To  find  the  exact  capacity,  we  have  that  -^l)  -»  .81,  -2/21  -*  14 

Ti  n'‘  ^  ’ 

“n 

■^3)  .05.  Applying  the  Theorem,  for  0  <  P  <  .81,  C^(P)  =  .405  log 

For  .81  <  P  <  (.81) (4)  +  (.14) (3)  =3.66,  we  have 


P  +  .81 


.81 


C^(P)  =  .405  log 
If  P  >  3.66,  then 

C^(P)  =  .405  log 


P  +  .81  +  .28 


.95 


P  +  1.09  +  .2 


1 


+  .07  log 


+  .07  log 


P  +  1.09 


U.95)(2)J 


P  +  1.29 


+  . 025  1 og 


P  +  1.29 
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M 


It  may  be  noted  that  the  value  Rp^(i,i)  =  4  +  i  ,  which  occurs  only  a  finite 
number  of  times,  plays  no  part  in  coding  capacity.  Neither  does  the  limit 

“n 

point  6^  =  9,  which  has  ^(4)  0  as  n  However,  the  bounds  given  by  the 

Proposition  do  not  take  into  account  the  "relative  frequencies"  lim  M^fiVn 

n 

so  that  those  bounds  can  be  quite  poor.  In  the  above  example,  this  causes  the 
lower  bound  given  by  the  Proposition,  which  uses  0^  =  9,  to  be  poor. 


» 
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